Gpu usage info and pid of workers in model description API #680

AshwinChafale · 2020-09-10T13:49:37Z

Description

Currently, the Model Description API gives no GPU usage info nor the PID of workers, which is not convenient for those who want to manage the status of running models. So in this PR, I have added the gpu-usage and pid of workers fields in the Model Description API.

Fixes #597

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Test Logs

Following are the test logs,

ubuntu@ip-172-31-41-36:~$ curl http://localhost:8081/models/densenet161
[
  {
    "modelName": "densenet161",
    "modelVersion": "1.0",
    "modelUrl": "densenet161.mar",
    "runtime": "python",
    "minWorkers": 4,
    "maxWorkers": 4,
    "batchSize": 1,
    "maxBatchDelay": 100,
    "loadedAtStartup": true,
    "workers": [
      {
        "id": "9000",
        "startTime": "2020-09-10T12:55:20.122Z",
        "status": "READY",
        "gpu": true,
        "memoryUsage": 3454746624,
        "pid": 9083,
        "gpuUsage": "gpuId::1 utilization.gpu [%]::0 %   utilization.memory [%]::0 %   memory.used [MiB]::1322 MiB "
      },
      {
        "id": "9001",
        "startTime": "2020-09-10T12:55:20.124Z",
        "status": "READY",
        "gpu": true,
        "memoryUsage": 3434995712,
        "pid": 9081,
        "gpuUsage": "gpuId::2 utilization.gpu [%]::0 %   utilization.memory [%]::0 %   memory.used [MiB]::1322 MiB "
      },
      {
        "id": "9002",
        "startTime": "2020-09-10T12:55:20.124Z",
        "status": "READY",
        "gpu": true,
        "memoryUsage": 3452321792,
        "pid": 9082,
        "gpuUsage": "gpuId::3 utilization.gpu [%]::0 %   utilization.memory [%]::0 %   memory.used [MiB]::1322 MiB "
      },
      {
        "id": "9003",
        "startTime": "2020-09-10T12:55:20.124Z",
        "status": "READY",
        "gpu": true,
        "memoryUsage": 2712588288,
        "pid": 9080,
        "gpuUsage": "gpuId::0 utilization.gpu [%]::0 %   utilization.memory [%]::0 %   memory.used [MiB]::5255 MiB "
      }
    ]
  }
]

Merging the current changes

sagemaker-neo-ci-bot · 2020-09-10T13:52:27Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: dc3e363
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-10T13:53:21Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: dc3e363
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-10T14:02:54Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 2a5e0c1
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-10T14:03:17Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 2a5e0c1
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-10T14:10:22Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 1b113aa
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-10T14:10:54Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 1b113aa
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Merging changes from origin repo

sagemaker-neo-ci-bot · 2020-09-10T14:30:55Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 34072d2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-10T14:31:09Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 34072d2
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…i error

…to issue_597

sagemaker-neo-ci-bot · 2020-09-11T06:11:00Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 78c8bf7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-11T06:11:25Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 78c8bf7
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

maaquib

It would also be useful to have the GPU usage info added to the metrics

sagemaker-neo-ci-bot · 2020-09-29T19:30:50Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 23a2a74
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-29T19:31:11Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 23a2a74
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…ad.java Co-authored-by: Aaqib <maaquib@gmail.com>

sagemaker-neo-ci-bot · 2020-09-30T11:47:08Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 5c6b55a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-30T11:47:41Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 5c6b55a
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-30T12:12:54Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: a03dfe5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-09-30T12:13:18Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: a03dfe5
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-10-14T09:12:50Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 9408b31
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-10-14T09:18:18Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 9408b31
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

harshbafna · 2020-10-14T12:14:40Z

It would also be useful to have the GPU usage info added to the metrics

@maaquib This can be taken up independently, as this will required changes in the metric logging code.

sagemaker-neo-ci-bot · 2020-10-14T12:18:44Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 90a4900
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-10-14T12:23:23Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 90a4900
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-10-14T12:32:35Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 07a0331
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-10-14T12:38:03Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 07a0331
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-10-21T17:40:08Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 22a5c58
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-10-21T17:45:44Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 22a5c58
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Outdated

sagemaker-neo-ci-bot · 2020-10-22T17:36:47Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 22a5c58
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2020-10-22T17:42:21Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 22a5c58
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AshwinChafale and others added 13 commits August 14, 2020 15:26

Merge pull request #1 from pytorch/master

27e4f1d

Merging the current changes

Added environment header in Sanity & Regression test suite

992fbfc

Added collect_env.py python script

34ca168

Undo few unwanted changes

56db699

Added cuda memory usage info and pid of workers in model description API

daea579

Modified WorkerThread.java file

2fcedca

.

963f26c

.

ef892d7

.

6fa04ef

..

7027ae9

Updated WorkerThread.java file

44bcd6d

Show gpuUsage specific to gpu-id on which the model is loaded

7b4e4f7

Deleted cuda_script.py file

dc3e363

AshwinChafale added 2 commits September 10, 2020 19:29

Fixed sanity failed log : Array brackets at illegal position.

2a5e0c1

Fixed sanity failed log : Array brackets at illegal position.

1b113aa

Merge pull request #2 from pytorch/master

34072d2

Merging changes from origin repo

AshwinChafale added 2 commits September 11, 2020 11:31

Updated WorkerThread.java file to handle Cannot run program nvidia-sm…

be5246c

…i error

Merge branch 'issue_597' of https://github.com/AshwinChafale/serve in…

78c8bf7

…to issue_597

maaquib requested review from codinnvrends and harshbafna September 11, 2020 17:27

maaquib suggested changes Sep 29, 2020

View reviewed changes

Update frontend/server/src/main/java/org/pytorch/serve/wlm/WorkerThre…

5c6b55a

…ad.java Co-authored-by: Aaqib <maaquib@gmail.com>

Updated server/src/main/java/org/pytorch/serve/wlm/WorkerThread.java

a03dfe5

chauhang requested a review from HamidShojanazeri October 6, 2020 16:05

Merge branch 'master' into issue_597

9408b31

added todo

90a4900

fixed java formatting

07a0331

maaquib approved these changes Oct 20, 2020

View reviewed changes

maaquib requested a review from codinnvrends October 20, 2020 17:48

Merge branch 'master' into issue_597

22a5c58

dk19y approved these changes Oct 21, 2020

View reviewed changes

maaquib merged commit f73ba66 into pytorch:master Oct 22, 2020

Gpu usage info and pid of workers in model description API #680

Gpu usage info and pid of workers in model description API #680

Uh oh!

Conversation

AshwinChafale commented Sep 10, 2020

Description

Type of change

Test Logs

Uh oh!

sagemaker-neo-ci-bot commented Sep 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 10, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 11, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 11, 2020

AWS CodeBuild CI Report

Uh oh!

maaquib left a comment

Choose a reason for hiding this comment

Uh oh!

sagemaker-neo-ci-bot commented Sep 29, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 29, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 30, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 30, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 30, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Sep 30, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Oct 14, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Oct 14, 2020

AWS CodeBuild CI Report

Uh oh!

harshbafna commented Oct 14, 2020

Uh oh!

sagemaker-neo-ci-bot commented Oct 14, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Oct 14, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Oct 14, 2020

AWS CodeBuild CI Report

Uh oh!

sagemaker-neo-ci-bot commented Oct 14, 2020

AWS CodeBuild CI Report

Uh oh!